Consistent Preordering with an Estimated Criterion Function, with an Application to the Evaluation and Comparison of Volatility Models

It is well known that the use of different criteria for estimation and evaluation can affect the ranking of models. This paper uncovers a different problem – and inconsistency – that can arise when alternatives are compared using an estimated criterion function. Our results are relevant for empirical studies where evaluation is based on estimated criteria functions, such as utility functions of heterogeneous individuals and forecast evaluation in a situation where the realized values of the target are observed with noise. We consider a general framework where stochastic sequences are ranked according to expected loss, using a parametric loss function. Besides the true preordering an empirical preordering and its asymptotic limit are of interest. We show that these preorderings are not equivalent in general and provide conditions that ensure the equivalence. We apply the framework to out-of-sample comparisons of ARCH-type models, where the need to substitute a proxy for the unobserved conditional variance, σ t , t = 1, 2, . . . can induce an inconsistency in the ranking of models. The reason is that the measurement error of σ t can distort the ranking – even as the sample size increases. The practical relevance of our theoretical results is accentuated by the empirical findings and a simulation study. In the context of volatility models, our results provide an additional argument for using intra-day data to approximate σ t , such as realized volatility. JEL Classification: C22; C52; C53; D0;